linux-kernel - Re: [PATCH] bcachefs: Switch to memalloc_flags

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CALOAHbCPiDbQuSzZE-9VuHwkjX-UfXsHPMfdowdG31KhTPMXPQ@mail.gmail.com>
Date: Mon, 2 Sep 2024 11:00:22 +0800
From: Yafang Shao <laoar.shao@...il.com>
To: Vlastimil Babka <vbabka@...e.cz>
Cc: Dave Chinner <david@...morbit.com>, Kent Overstreet <kent.overstreet@...ux.dev>, 
	Michal Hocko <mhocko@...e.com>, Matthew Wilcox <willy@...radead.org>, linux-fsdevel@...r.kernel.org, 
	linux-mm@...ck.org, linux-kernel@...r.kernel.org, 
	Dave Chinner <dchinner@...hat.com>
Subject: Re: [PATCH] bcachefs: Switch to memalloc_flags_do() for vmalloc allocations

On Fri, Aug 30, 2024 at 11:25 PM Vlastimil Babka <vbabka@...e.cz> wrote:
>
> On 8/30/24 11:14, Yafang Shao wrote:
> > On Thu, Aug 29, 2024 at 10:29 PM Dave Chinner <david@...morbit.com> wrote:
> >
> > Hello Dave,
> >
> > I've noticed that XFS has increasingly replaced kmem_alloc() with
> > __GFP_NOFAIL. For example, in kernel 4.19.y, there are 0 instances of
> > __GFP_NOFAIL under fs/xfs, but in kernel 6.1.y, there are 41
> > occurrences. In kmem_alloc(), there's an explicit
> > memalloc_retry_wait() to throttle the allocator under heavy memory
> > pressure, which aligns with your filesystem design. However, using
> > __GFP_NOFAIL removes this throttling mechanism, potentially causing
> > issues when the system is under heavy memory load. I'm concerned that
> > this shift might not be a beneficial trend.
> >
> > We have been using XFS for our big data servers for years, and it has
> > consistently performed well with older kernels like 4.19.y. However,
> > after upgrading all our servers from 4.19.y to 6.1.y over the past two
> > years, we have frequently encountered livelock issues caused by memory
> > exhaustion. To mitigate this, we've had to limit the RSS of
> > applications, which isn't an ideal solution and represents a worrying
> > trend.
>
> By "livelock issues caused by memory exhaustion" you mean the long-standing
> infamous issue that the system might become thrashing for the remaining
> small amount of page cache, and anonymous memory being swapped out/in,
> instead of issuing OOM, because there's always just enough progress of the
> reclaim to keep going, but the system isn't basically doing anything else?
>

Exactly

> I think that's related to near-exhausted memory by userspace,

If user space is the root cause, the appropriate response should be to
terminate the offending user tasks. However, this doesn't happen at
all.

> so I'm not
> sure why XFS would be to blame here.

Honestly, I'm not sure what to blame, as I don't have a clear
understanding of what's happening during memory allocation. One server
among tens of thousands in production randomly experiences a livelock
within days, making it extremely difficult to pinpoint the root cause.

>
> That said, if memalloc_retry_wait() is indeed a useful mechanism, maybe we
> could perform it inside the page allocator itself for __GFP_NOFAIL?

Perhaps an additional wait or exit mechanism should be implemented
specifically for __GFP_NOFAIL.

--
Regards
Yafang