Message-ID: <C69EF2FE-DFF6-492E-AD40-97A53739C3EC@vmware.com>
Date:   Tue, 2 Nov 2021 08:48:27 +0000
From:   Alexey Makhalov <amakhalov@...are.com>
To:     David Hildenbrand <david@...hat.com>,
        Michal Hocko <mhocko@...e.com>
CC:     "linux-mm@...ck.org" <linux-mm@...ck.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "stable@...r.kernel.org" <stable@...r.kernel.org>,
        Oscar Salvador <OSalvador@...e.com>
Subject: Re: [PATCH] mm: fix panic in __alloc_pages



On 11/2/21, 1:12 AM, "David Hildenbrand" <david@...hat.com> wrote:

Thanks for reviews,

    On 02.11.21 08:47, Michal Hocko wrote:
    > [CC Oscar and David]
    > 
    > On Mon 01-11-21 13:13:12, Alexey Makhalov wrote:
    >> There is a kernel panic caused by __alloc_pages() accessing
    >> uninitialized NODE_DATA(nid). Uninitialized node data exists
    >> during the time when CPU with memoryless node was added but
    >> not onlined yet. Panic can be easy reproduced by disabling
    >> udev rule for automatic onlining hot added CPU followed by
    >> CPU with memoryless node hot add.
    >>
    >> This is a panic caused by percpu code doing allocations for
    >> all possible CPUs and hitting this issue:
    >>
    >>  CPU2 has been hot-added
    >>  BUG: unable to handle page fault for address: 0000000000001608
    >>  #PF: supervisor read access in kernel mode
    >>  #PF: error_code(0x0000) - not-present page
    >>  PGD 0 P4D 0
    >>  Oops: 0000 [#1] SMP PTI
    >>  CPU: 0 PID: 1 Comm: systemd Tainted: G            E     5.15.0-rc7+ #11
    >>  Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW
    >>
    >>  RIP: 0010:__alloc_pages+0x127/0x290
    > 
    > Could you resolve this into a specific line of the source code please?
    > 
    >>  Code: 4c 89 f0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 44 89 e0 48 8b 55 b8 c1 e8 0c 83 e0 01 88 45 d0 4c 89 c8 48 85 d2 0f 85 1a 01 00 00 <45> 3b 41 08 0f 82 10 01 00 00 48 89 45 c0 48 8b 00 44 89 e2 81 e2
    >>  RSP: 0018:ffffc900006f3bc8 EFLAGS: 00010246
    >>  RAX: 0000000000001600 RBX: 0000000000000000 RCX: 0000000000000000
    >>  RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000cc2
    >>  RBP: ffffc900006f3c18 R08: 0000000000000001 R09: 0000000000001600
    >>  R10: ffffc900006f3a40 R11: ffff88813c9fffe8 R12: 0000000000000cc2
    >>  R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000cc2
    >>  FS:  00007f27ead70500(0000) GS:ffff88807ce00000(0000) knlGS:0000000000000000
    >>  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    >>  CR2: 0000000000001608 CR3: 000000000582c003 CR4: 00000000001706b0
    >>  Call Trace:
    >>   pcpu_alloc_pages.constprop.0+0xe4/0x1c0
    >>   pcpu_populate_chunk+0x33/0xb0
    >>   pcpu_alloc+0x4d3/0x6f0
    >>   __alloc_percpu_gfp+0xd/0x10
    >>   alloc_mem_cgroup_per_node_info+0x54/0xb0
    >>   mem_cgroup_alloc+0xed/0x2f0
    >>   mem_cgroup_css_alloc+0x33/0x2f0
    >>   css_create+0x3a/0x1f0
    >>   cgroup_apply_control_enable+0x12b/0x150
    >>   cgroup_mkdir+0xdd/0x110
    >>   kernfs_iop_mkdir+0x4f/0x80
    >>   vfs_mkdir+0x178/0x230
    >>   do_mkdirat+0xfd/0x120
    >>   __x64_sys_mkdir+0x47/0x70
    >>   ? syscall_exit_to_user_mode+0x21/0x50
    >>   do_syscall_64+0x43/0x90
    >>   entry_SYSCALL_64_after_hwframe+0x44/0xae
    >>
    >> Node can be in one of the following states:
    >> 1. not present (nid == NUMA_NO_NODE)
    >> 2. present, but offline (nid > NUMA_NO_NODE, node_online(nid) == 0,
    >> 				NODE_DATA(nid) == NULL)
    >> 3. present and online (nid > NUMA_NO_NODE, node_online(nid) > 0,
    >> 				NODE_DATA(nid) != NULL)
    >>
    >> alloc_page_{bulk_array}node() functions verify for nid validity only
    >> and do not check if nid is online. Enhanced verification check allows
    >> to handle page allocation when node is in 2nd state.
    > 
    > I do not think this is a correct approach. We should make sure that the
    > proper fallback node is used instead. This means that the zone list is
    > initialized properly. IIRC this has been a problem in the past and it
    > has been fixed. The initialization code is quite subtle though so it is
    > possible that this got broken again.
This approach behaves in the same way as CPU was not yet added. (state #1).
So, we can think of state #2 as state #1 when CPU is not present.

    I'm a little confused:

    In add_memory_resource() we hotplug the new node if required and set it
    online. Memory might get onlined later, via online_pages().
You are correct. In case of memory hot add, it is true. But in case of adding
CPU with memoryless node, try_node_online() will be called only during CPU
onlining, see cpu_up().

Is there any reason why try_online_node() resides in cpu_up() and not in add_cpu()?
I think it would be correct to online node during the CPU hot add to align with
memory hot add.

    So after add_memory_resource()->__try_online_node() succeeded, we have
    an online pgdat -- essentially 3.

    This patch detects if we're past 3. but says that it reproduced by
    disabling *memory* onlining.
This is the hot adding of both new CPU and new _memoryless_ node (with CPU only)
And onlining CPU makes its node online. Disabling CPU onlining puts new node
into state #2, which leads to repro.    

    Before we online memory for a hotplugged node, all zones are !populated.
    So once we online memory for a !populated zone in online_pages(), we
    trigger setup_zone_pageset().


    The confusing part is that this patch checks for 3. but says it can be
    reproduced by not onlining *memory*. There seems to be something missing.

    Do we maybe need a proper populated_zone() check before accessing zone data?

Thanks,
--Alexey

Powered by blists - more mailing lists