linux-kernel - Re: [RFC PATCH 0/2] mm: fix OOMs for binding workloads to movable zone only node

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20201104073826.GA15700@shbuild999.sh.intel.com>
Date:   Wed, 4 Nov 2020 15:38:26 +0800
From:   Feng Tang <feng.tang@...el.com>
To:     Michal Hocko <mhocko@...e.com>
Cc:     Andrew Morton <akpm@...ux-foundation.org>,
        Johannes Weiner <hannes@...xchg.org>,
        Matthew Wilcox <willy@...radead.org>,
        Mel Gorman <mgorman@...e.de>, dave.hansen@...el.com,
        ying.huang@...el.com, linux-mm@...ck.org,
        linux-kernel@...r.kernel.org
Subject: Re: [RFC PATCH 0/2] mm: fix OOMs for binding workloads to movable
 zone only node

Hi Michal,

Thanks for the prompt review!

On Wed, Nov 04, 2020 at 08:13:08AM +0100, Michal Hocko wrote:
> On Wed 04-11-20 14:10:08, Feng Tang wrote:
> > Hi,
> > 
> > This patchset tries to report a problem and get suggestion/review
> > for the RFC fix patches.
> > 
> > We recently got a OOM report, that when user try to bind a docker(container)
> > instance to a memory node which only has movable zones, and OOM killing
> > still can't solve the page allocation failure.
> 
> This is a cpuset node binding right?

Yes.

A simple test command is 'docker run -it --cpuset-mems 3 ubuntu:latest'
while the node 3 is a movable only PMEM node.
 
> > The callstack was:
> > 
> > 	[ 1387.877565] runc:[2:INIT] invoked oom-killer: gfp_mask=0x500cc2(GFP_HIGHUSER|__GFP_ACCOUNT), order=0, oom_score_adj=0
> > 	[ 1387.877568] CPU: 8 PID: 8291 Comm: runc:[2:INIT] Tainted: G        W I E     5.8.2-0.g71b519a-default #1 openSUSE Tumbleweed (unreleased)
> > 	[ 1387.877569] Hardware name: Dell Inc. PowerEdge R640/0PHYDR, BIOS 2.6.4 04/09/2020
> > 	[ 1387.877570] Call Trace:
> > 	[ 1387.877579]  dump_stack+0x6b/0x88
> > 	[ 1387.877584]  dump_header+0x4a/0x1e2
> > 	[ 1387.877586]  oom_kill_process.cold+0xb/0x10
> > 	[ 1387.877588]  out_of_memory.part.0+0xaf/0x230
> > 	[ 1387.877591]  out_of_memory+0x3d/0x80
> > 	[ 1387.877595]  __alloc_pages_slowpath.constprop.0+0x954/0xa20
> > 	[ 1387.877599]  __alloc_pages_nodemask+0x2d3/0x300
> > 	[ 1387.877602]  pipe_write+0x322/0x590
> > 	[ 1387.877607]  new_sync_write+0x196/0x1b0
> > 	[ 1387.877609]  vfs_write+0x1c3/0x1f0
> > 	[ 1387.877611]  ksys_write+0xa7/0xe0
> > 	[ 1387.877617]  do_syscall_64+0x52/0xd0
> > 	[ 1387.877621]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > 
> > The meminfo log only shows the movable only node, which has plenty
> > of free memory. And in our reproducing with 1/2 patch, the normal
> > node (has DMA/DMA32/Normal) also has lot of free memory when OOM
> > happens. 
> 
> OK, so you are bidning to a movable node only and your above request is
> for GFP_HIGHUSER which _cannot_ be satisfied from the movable zones
> because that memory is not movable. So the system behaves as expected.
> Your cpuset is misconfigured IMHO. Movable only nodes come with their
> risk and configuration price.

Aha, this is what we told the reporter at first. Their platform is 2S
platform, and each socket has one DRAM node + one persistent memory node,
and we suggested to bind the docker to one DRAM + one PMEM node.

> > If we hack to make this (GFP_HIGHUSER|__GFP_ACCOUNT) request get
> > a page, and following full docker run (like installing and running
> > 'stress-ng' stress test) will see more allocation failures due to
> > different kinds of request(gfp_masks). And the 2/2 patch will detect
> > such cases that the allowed target nodes only have movable zones
> > and loose the binding check, otherwise it will trigger OOM while
> > the OOM won't do any help, as the problem is not lack of free memory.
> 
> Well, this breaks the cpuset containment, right? I consider this quite
> unexpected for something that looks like a misconfiguration. I do agree
> that this is unexpected for anybody who is not really familiar with
> concept of movable zone but we should probably call out all these
> details rather than tweak the existing semantic.

Yes, it does break the cpuset containment 

> Could you be more specific about the usecase here? Why do you need a
> binding to a pure movable node? 

One common configuration for a platform is small size of DRAM plus huge
size of PMEM (which is slower but cheaper), and my guess of their use
is to try to lead the bulk of user space allocation (GFP_HIGHUSER_MOVABLE)
to PMEM node, and only let DRAM be used as less as possible. 

Thanks,
Feng

> -- 
> Michal Hocko
> SUSE Labs